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INTRODUCTION 


In  1989,  Alberta  Education  undertook  a three-year  study  to  find  out  if  language  of  testing 
is  a variable  that  affects  the  way  French  Immersion  students  respond  to  test  questions. 
This  was  part  of  a larger  discussion  of  the  issue  of  assessment  of  French  Immersion 
students  that  addressed  two  basic  questions: 

1.  In  what  way  can  the  achievement  of  French  Immersion  students  best  be  assessed? 

2.  How  can  the  results  be  reported  and  interpreted  most  validly  and  reliably? 

The  first  part  of  the  study,  conducted  in  1989,  was  reported  in  Provincial  Assessment  of 
Students  in  French  Immersion  Programs,  Special  Report,  June  1990.  The  second  part  of 
the  study,  conducted  in  1990,  was  reported  in  Language  of  Testing  Study  Report,  January 
1991.  These  reports  should  be  consulted  for  more  detailed  statements  of  the  issues  in 
question,  for  the  results  of  the  first  two  years  of  the  study,  and  for  a discussion  of  these 
results. 

The  purpose  of  this  paper  is  to  present  the  design  of  the  1991  study,  the  results  obtained,  a 
comparison  of  the  1989,  1990,  and  1991  results,  and  a discussion  of  the  results  and  the 
related  issues.  We  believe  that  this  will  help  in  determining  the  best  course  of  action  for 
assessment  of  French  Immersion  students. 
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DESIGN  OF  THE  1991  STUDY 


The  1991  study  dealt  with  the  scheduled  provincial  achievement  tests:  Grade  3 Science, 
Grade  6 Mathematics,  and  Grade  9 Social  Studies.  Schools  offering  French  Immersion 
programs  at  the  grades  3,  6,  or  9 levels  were  randomly  selected  for  the  study.  The  sample 
contained  38  schools  from  23  different  jurisdictions.  There  was  a balance  of 
representation  from  large  and  small  schools,  and  from  urban  and  rural  jurisdictions. 

The  school  superintendents  concerned  were  contacted  to  obtain  permission  to  have  some 
of  the  students  from  their  jurisdiction  participate  in  this  study.  The  study  design  and 
procedure  for  reporting  results  were  presented  and  class  lists  were  requested  at  this 
time. 

Using  the  class  lists  from  participating  schools,  staff  from  the  Student  Evaluation 
Branch  randomly  assigned  English  or  French  forms  of  the  appropriate  achievement  test 
to  equal  numbers  of  students  in  each  classroom.  These  tests  were  administered  at  the 
date  and  time  regularly  scheduled  for  provincial  achievement  tests. 

The  returned  tests  were  scored  under  the  direction  of  Student  Evaluation  Branch  staff. 
The  results  were  then  analyzed  to  determine  if  there  were  significant  differences  in  the 
scores  achieved  by  the  two  groups  of  students. 


Grade  3 Science 

In  June  1991,  466  Grade  3 French  Immersion  students  from  14  different  schools  in  12 
jurisdictions  wrote  either  the  English  version  or  a French  translation  of  the  1991  Grade  3 
Science  Achievement  Test.  All  students  taking  part  in  the  study  had  received  their 
Grade  3 science  instruction  in  French. 


Grade  6 Mathematics 

In  June  1991,  576  Grade  6 French  Immersion  students  from  16  different  schools  in  13 
jurisdictions  wrote  either  the  English  version  or  a French  translation  of  the  1991  Grade  6 
Mathematics  Achievement  Test.  All  students  taking  part  in  the  study  had  received  their 
Grade  6 Mathematics  instruction  in  French. 


Grade  9 Social  Studies 

In  June  1991,  517  Grade  9 French  Immersion  students  from  10  different  schools  in  8 
jurisdictions  wrote  either  the  English  version  or  a French  translation  of  the  1991  Grade  9 
Social  Studies  Achievement  Test.  All  students  taking  part  in  the  study  had  received  their 
Grade  9 Social  Studies  instruction  in  French. 
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RESULTS  OF  THE  1991  STUDY 


Grade  3 Science 

Table  1 presents  a comparison,  by  reporting  category,  of  the  scores  achieved  by  the 
students  in  the  Language  of  Testing  Study  on  the  English  and  French  forms  of  the  1991 
Grade  3 Science  Achievement  Test.  The  results  indicate  that  the  group  of  students  who 
wrote  the  test  in  French  (LOTF)  achieved  significantly  lower  scores  than  those  who  wrote 
the  test  in  English  (LOTE)  in  all  reporting  categories.  These  results  support  the 
hypothesis  that,  in  Grade  3,  the  responses  of  French  Immersion  students  are  sensitive  to 
the  language  of  testing. 

When  scores  achieved  by  French  Immersion  students  on  the  English  test  form  (LOTE) 
were  compared  with  the  1991  provincial  averages,  no  significant  difference  was  found. 
This  suggests  that  French  Immersion  students  are  able  to  transfer  and  apply  the  skills 
and  knowledge  they  have  gained  while  studying  in  French  to  an  English  language 
situation  and  perform  about  as  well  as  the  general  population  writing  the  test  in  English. 

Scores  achieved  by  French  Immersion  students  on  the  French  form  of  the  test  (LOTF) 
were  found  to  be  significantly  lower  than  the  1991  provincial  average  in  all  reporting 
categories.  Keeping  in  mind  that  the  466  Grade  3 students  who  were  part  of  this  study 
were  randomly  assigned  to  write  the  English  or  the  French  form  of  the  test,  we  can 
assume  that  the  two  groups  of  students  were  approximately  equal  in  terms  of  knowledge 
and  skills  in  science.  The  lower  scores  on  the  French  form  of  the  test  show  that  these 
students  do  not  yet  possess  a sufficiently  strong  command  of  the  French  language  to  be 
able  to  show  adequately  what  they  know  and  can  do  in  science  when  they  write  a science 
test  in  French. 
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Table  1 

Language  of  Testing  (LOT)  Study  1991 
Results  by  Reporting  Category  for  Grade  3 Science 


The  bracketed  figures  below  the  LOTE  group  means  are  the  1991  Grade  3 provincial 
averages  for  students  in  the  regular  program. 


*S.D.  - Standard  Deviation 

**t  - a test  statistic  for  determining  the  significance  of  difference  between  two  test  means 
***Prob.  - Probability  levels  less  than  .01  are  considered  significant 
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Grade  6 Mathematics 

Table  2 presents  a comparison,  by  reporting  category,  of  the  scores  achieved  by  the 
students  in  the  Language  of  Testing  Study  on  the  English  and  French  forms  of  the  June 
1991  Grade  6 Mathematics  Achievement  Test.  The  results  indicate  that  the  group  of 
students  who  wrote  the  French  version  of  the  test  (LOTF)  achieved  lower  scores  than  did 
the  students  who  wrote  the  test  in  English  (LOTE)  on  all  reporting  categories.  That 
difference  was  significant  (probability  <.01 ) on  the  total  test,  on  the  Process  Skills 
component,  on  three  of  the  six  curricular  elements,  and  on  the  Application/Problem 
Solving  cognitive  level.  The  difference  was  not  significant  (probability  >.01)  on  the  Subject 
Matter  component,  three  of  the  six  curricular  elements,  and  on  the  Knowledge  and 
Comprehension  cognitive  levels. 

When  scores  achieved  by  the  French  Immersion  students  on  the  English  form  of  the  test 
were  compared  to  the  1991  provincial  averages,  results  showed  that  students  in  French 
Immersion  exceeded  the  provincial  averages  in  all  reporting  categories.  This  suggests 
that  the  achievement  of  students  in  French  Immersion  is  somewhat  higher  than  that  of 
students  in  the  English  program. 

These  results  show  that  French  Immersion  students  develop  mathematical  knowledge, 
understanding,  and  skills  as  evidenced  by  their  scores  on  both  the  English  and  the 
French  forms  of  the  test. 
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Table  2 

Language  of  Testing  Study  (LOT)  1991 
Results  by  Reporting  Category  for  Grade  6 Mathematics 


*S.D.  - Standard  Deviation 

**t  - a test  statistic  for  determining  the  significance  of  difference  between  two  test  means 
***Prob.  - Probability  levels  less  than  .01  are  considered  significant 
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Grade  9 Social  Studies 


Tables  3 to  6 present  a comparison,  by  reporting  category,  of  the  scores  achieved  by  the 
students  in  the  Language  of  Testing  Study  who  wrote  the  English  and  the  French  forms 
of  the  1991  Social  Studies  Achievement  Test.  The  results  show  that  the  group  of  students 
who  wrote  the  test  in  English  (LOTE)  achieved  significantly  higher  scores  than  those  who 
wrote  the  test  in  French  (LOTF)  on  all  reporting  categories  except  for  Topic  A.  These 
results  support  the  hypothesis  that  in 

Grade  9,  the  responses  of  French  Immersion  students  are  sensitive  to  the  language  of 
testing. 

When  scores  achieved  by  French  Immersion  students  on  the  English  form  of  the  test  were 
compared  with  the  1991  provincial  averages,  results  showed  that  students  in  French 
Immersion  programs  exceeded  the  provincial  averages  in  all  reporting  categories  for 
Grade  9 Social  Studies. 

When  the  scores  achieved  by  French  Immersion  students  on  the  French  form  of  the  test 
were  compared  with  the  1991  provincial  average,  results  showed  that  students  in  French 
Immersion  programs  equaled  or  slightly  exceeded  the  provincial  average  in  all  reporting 
categories. 
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Table  3 

Language  of  Testing  (LOT)  Study  1991 
Multiple-Choice  Results  by  Reporting  Category  for  Grade  9 Social  Studies 


Reporting 
! Category 

...  ..  . 

LOTE 

Mean 

(N=259) 

LOTF 

Mean 

(N=258) 

LOTE 

S.D.* 

LOTF 

S.D. 

t** 

Prob.***  11 

TOTAL  TEST 

42.3 

(37.4) 

PjjgM ; 

8.7 

4.60 

.000 

Topic  A:  Economic 
Growth:  U.S.A. 

14.0 

(12.6) 

■■111 

3.3 

1.29 

.199 

Topic  B:  Economic 
Growth:  U.S.S.R. 

13.3 

(11.8) 

1 U8 

3.3 

4.84 

.000 

Topic  C:  Canada: 
Responding  to  Change 

15.0 

(13.0) 

m 

3.5 

fg  $3 

5.79 

.000 

Recall  of  Knowledge: 
Facts,  Concepts,  and 
Generalizations 

20.4 

(17.9) 

i 

{ 18.$ 

Sllill!!!!!! 

4.6 

4.6 

3.86 

.000 

Application  of  Process 
Skills 

21.9 

(19.4) 

: i9.8 

4.8 

\ 56 

4.72 

.000 

Recall  of  Knowledge: 

^ . 

li;  . 

fy. 

Topic  A 

6.4 

(5.7) 

\ €.6 
6,1 

1.7 

1 13 

-1.11 

.270  1 

Topic  B 

7.1 

(6.3) 

2.0 

> SUL 

5.34 

.000 

Topic  C 

6.9 

(6.0) 

, 6.1 

fi.,  . S 

1.9 

l L9 

I;-  - • 1 i 

4.72 

.000 

! 

Application  of  Process 
Skills: 

I8SI11II 

I/  J 

Locating,  Interpreting, 
l and  Organizing 

10.7 

(9.4) 

2.6 

|(:vao 

fc 

3.92 

.000 

Analysing, 
Synthesizing,  and 
Evaluating 

11.2 

(10.0) 

1 3M  | 
* 

2.7 

P1  . 

4.55 

rJ 

- ■ ■ ! — ! ' viVmvm mYiVmVi aasaai  1 gaSaSai V'V|V 'Vi — 

The  bracketed  figures  below  the  LOTE  group  means  are  the  1991  Grade  9 provincial 
averages  for  students  in  the  regular  program. 


*S.D.  - Standard  Deviation 

**t  - a test  statistic  for  determining  the  significance  of  difference  between  two  test  means 

***Prob.  - Probability  levels  less  than  .01  are  considered  significant 
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Table  4 

Language  of  Testing  (LOT)  Study  1991 
Distribution  of  Written-Response  Scale  Points  by  Scoring  Category 
for  Grade  9 Social  Studies 


Support  of  Position 

Expression  | 

Scale  Points 

LOTE 

LOTF 

LOTE 

LOTF 

(Per  cent) 

(Per  cent) 

(Per  cent) 

(Per  cent)  I 

5 (Excellent) 

5.8 

4.7 

9.3 

3.5  jj! 

(3.8) 

(3.7) 

| 

4 (Good) 

26.3 

29.1 

29.0 

26.7  I 

(15.9) 

(19.1) 

1 

3 (Acceptable) 

45.6 

46.9 

49.4 

49.2  | 

(42.6) 

(52.6) 

2 (Limited) 

18.1 

16.3 

9.3 

19.4 

(29.0) 

(20.3) 

j 1 (Poor) 

42 

2.7 

3.1 

0.8 

(7.8) 

(3.4) 

Ins  (Insufficient) 

0.0 

0.4 

0.0 

0.4 

1 

(1.0) 

(1.0) 



The  bracketed  figures  below  the  LOTE  group  percentages  are  the  1991  Grade  9 provincial 
percentages  for  students  in  the  regular  program. 


Note:  Chi-Square  for  Support  of  Position  = 2.88  (n.s.) 

Chi-Square  for  Expression  = 20.8  (pc. 001) 
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Table  5 

Language  of  Testing  (LOT)  Study 

Means  for  Written-Response  by  Scoring  Category  for  Grade  9 Social  Studies 


Support  of  Position 
(points  out  of  20) 

Expression 
(points  out  of  10) 

LOTE 

LOTE 

LOTE 

LOTE 

12.4 

12.6 

6.2 

6.3 

(11.0) 

(5.5) 

The  bracketed  figures  below  the  LOTE  groups  are  the  1991  Grade  9 provincial  figures  for 
students  in  the  regular  program. 


Tahle6 

Language  of  Testing  (LOT)  Study 

Total  Written-Response  Means  for  Grade  9 Social  Studies  (points  out  of  30) 


LOTE 

LOTE 

LOTE 

LOTE 

t 

Prob. 

Mean 

Mean 

SJX 

S.D. 

19.1 

18.9 

5.0 

4.7 

.56 

.578 

(17.0) 

(5.2) 

The  bracketed  figures  below  the  LOTE  groups  are  the  1991  Grade  9 provincial  figures  for 
students  in  the  regular  program. 
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COMPARISON  OF  1989, 1990,  AND  1991  LANGUAGE  OF  TESTING  STUDIES 


GradeS 

Results  of  the  1989, 1990,  and  1991  studies  show  that  student  responses  at  the  Grade  3 
level  are  sensitive  to  the  language  of  testing.  Student  responses  to  science,  mathematics, 
and  social  studies  questions  showed  that  French  Immersion  students  who  wrote  the 
English  form  of  the  achievement  test  achieved  significantly  higher  scores  than  French 
Immersion  students  who  wrote  the  French  translation  of  the  test.  The  difference 
between  the  two  groups  is  generally  greater  at  the  Grade  3 level  than  it  is  at  the  other  two 
levels. 


Grade  6 

Results  of  the  1989,  1990,  and  1991  studies  show  that  student  responses  at  the  Grade  6 
level  are  sensitive  to  the  language  of  testing.  Student  response  to  social  studies,  science, 
and  mathematics  questions  show  that  French  Immersion  students  who  wrote  the 
English  form  of  the  achievement  test  achieved  higher  scores  than  French  Immersion 
students  who  wrote  the  French  translation  of  the  test.  It  is  interesting  to  note  that  the 
differences  were  greatest  in  social  studies,  less  in  science,  and  least  in  mathematics. 


Grade  9 

Results  of  the  1991  study  show  that  student  responses  at  the  Grade  9 level  are  sensitive  to 
the  language  of  testing.  Student  responses  to 

Grade  9 social  studies  questions  showed  that  French  Immersion  students  who  wrote  the 
English  form  of  the  achievement  test  achieved  significantly  higher  scores  than  French 
Immersion  students  who  wrote  the  French  translation  of  the  test. 

Since  Grade  9 students  participated  in  only  one  year  of  study  we  cannot  easily  generalize, 
but  it  is  interesting  to  note  that  the  difference  is  considerably  smaller  than  the  difference 
at  the  lower  grades.  From  the  available  data,  we  cannot  tell  whether  this  is  due  to 
Immersion  students  having  a better  command  of  their  second  language  by  Grade  9 or 
whether  students  with  a weak  command  of  the  French  language  have  dropped  out  before 
Grade  9.  It  seems  reasonable  to  assume  that  both  of  these  factors  are  at  work  here. 


Summary  Tables 

Tables  7 to  10  compare  the  results  of  the  Language  of  Testing  study  year  by  year,  grade  by 
grade,  and  subject  by  subject. 
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Table  7 

Language  of  Testing  Study  - Test  Scores  by  Year 


1989 

LANGUAGE  OF  TESTING  STUDY 

Grade  Subject 

Test  Score 

English  Groups 

French  Groups 

Total  Possible 

3 

Social  Studies 

29.3 

(33.8)* 

21.8 

50 

6 

Social  Studies 

69.9 

(62.5) 

53.8 

[55.3]** 

100 

1990 

LANGUAGE  OF  TESTING  STUDY 

Grade  Subject 

Test  Score 

English  Groups 

French  Groups 

Total  Possible 

3 

Mathematics 

36.2 

(35.1)* 

32.3 

50 

6 

Science 

42.9 

(40.5) 

36.9 

[36.8]** 

60 

1991 

LANGUAGE  OF  TESTING  STUDY 

Grade  Subject 

Test  Score 

English  Groups 

French  Groups 

Total  Possible 

j 3 

Science 

33.9 

(34.2)* 

27.6 

50 

6 

Mathematics 

39.5 

(37.5) 

37.3 

[40.1]** 

55 

9 

Social  Studies 
(multiple  choice) 

42.3 

(37.4) 

38.6 

[36.8] 

60 

9 

Social  Studies 
(total  test) 

68.3 

(60.5) 

63.8 

[61.2] 

100 

*()  indicate  provincial  average 

**[]  indicate  average  for  total  French  Immersion  population;  in  1989,  the  score  includes 
both  French  Immersion  and  Francophone  student  results 
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Table  8 

Language  of  Testing  Study — Test  Scores  by  Grade 
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*Since  1989  was  not  a scheduled  achievement  testing  year  for  ^ English  Group  % 

Grade  3 Social  Studies,  the  1988  test  was  used  for  the  study.  g§  French  Group  % 

The  provincial  average  given  is  for  1988.  □ Provincial  Average  % 
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*Since  1989  was  not  a scheduled  achievement  testing  year  for  ^ English  Group  % 

Grade  3 Social  Studies,  the  1988  test  was  used  for  the  study.  §g  French  Group  % 

The  provincial  average  given  is  for  1988.  0 Provincial  Average  % 


Table  10 

Language  of  Testing  Study — Test  Scores  by  Subject 
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♦Since  1989  was  not  a scheduled  achievement  testing  year  for  ^ English  Group  % 

Grade  3 Social  Studies,  the  1988  test  was  used  for  the  study.  H§  French  Group  % 

The  provincial  average  given  is  for  1988.  0 Provincial  Average  % 


ENGLISH  LANGUAGE  ARTS  AND  FRENCH  LANGUAGE  ARTS 


Tests  designed  to  measure  achievement  in  English  Language  Arts  and  in  French 
Language  Arts  were  administered  as  shown  in  Table  11.  The  participation  of  French 
Immersion  students  in  the  French  Language  Arts  tests  was  at  the  option  of  the 
superintendent. 

Although  comparisons  should  be  done  only  with  caution,  it  is  of  some  interest  to  compare 
the  results  of  the  English  Language  Arts  test  with  those  of  the  French  Language  Arts 
test.  It  is  also  interesting  to  compare  the  English  achievement  scores  of  the  students  who 
took  the  French  test  with  the  average  for  all  those  who  took  the  English  test. 

Students  who  wrote  both  the  English  and  French  Language  Arts  tests  consistently 
obtained  higher  scores  on  the  English  Language  Arts  test  than  did  students  who  received 
instruction  in  English  only.  It  is  not  known,  of  course,  if  the  French  Immersion  students 
involved  would  have  achieved  higher  or  lower  English  Language  Arts  scores  if  they  had 
been  in  a regular  program.  There  is,  at  any  rate,  no  evidence  that  the  English  Language 
Arts  skills  of  students  taking  both  courses  are  any  less  than  the  skills  of  those  students 
taking  only  one  language  arts  course. 

Results  are  also  given  for  achievement  of  French  Immersion  students  on  the  French 
Language  Arts  Achievement  tests.  The  design  of  the  French  and  English  Language  Arts 
tests  were  parallel.  Results  show  that  students  enrolled  in  French  Immersion  programs 
also  perform  well  in  English  Language  Arts. 
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Table  11 


RESULTS  OF  FRENCH  IMMERSION  STUDENTS  FOR  FRENCH  LANGUAGE  ARTS 
AND  ENGLISH  LANGUAGE  ARTS  ACHIEVEMENT  TESTS 


Year 

Subject 

Mean 

1986 

Grade  9 French  Language  Arts 

62.8 

1986 

Grade  9 English  Language  Arts 

73.8 

(64.0)* 

1988 

Grade  6 French  Language  Arts 

65.7 

1988 

Grade  6 English  Language  Arts 

68.9 

(62.5)* 

1989** 

Grade  3 English  Language  Arts 

69.9 

(68.9)* 

1990 

Grade  9 French  Language  Arts 

63.2 

1990 

Grade  9 English  Language  Arts 

69.4 

(61.5)* 

*Denotes  provincial  average  for  all  students  who  wrote  the  English  Language  Arts  test 

**No  French  Language  Arts  test  is  developed  for  Grade  3 
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CONCLUSIONS 


The  purpose  of  the  1991  Language  of  Testing  Study  was  to  determine  if  a language  of 
testing  effect  existed  when  students  were  tested  in  French  or  in  English.  The  results 
show  that  performances  by  students  who  wrote  the  English  form  of  the  tests  were 
consistently  and  significantly  higher  than  were  the  performances  achieved  by  students 
who  wrote  the  tests  in  French. 

Because  schools  were  randomly  selected  and  students  were  randomly  assigned  to  the 
English  or  the  French  groups,  we  can  assume  that  all  conditions  (e.g.,  levels  of  ability, 
motivation,  quantity  and  quality  of  instruction)  except  the  language  of  testing  were  the 
same  across  the  two  groups  within  each  grade.  It  is  therefore  possible  to  attribute  the 
significant  differences  in  scores  across  groups  to  the  language  in  which  the  tests  were 
presented  and  written. 

Results  for  1991  were  significant  because  of  the  magnitude  of  the  language  of  testing 
effect  and  because  they  showed  the  same  patterns  of  differences  between  the  English  and 
French  groups  as  had  been  shown  on  the  1989  and  1990  studies.  Such  a variation  in 
scores  has  implications  for  data  interpretation.  Because  the  English  group  scores  were 
significantly  higher  than  the  provincial  averages,  a reasonable  interpretation  is  that 
students  in  French  Immersion  have  levels  of  achievement  that  are  higher  than  the 
provincial  levels. 

The  results  also  show  that  while  French  Immersion  students  as  a whole  have  a good 
command  of  subject  matter,  they  are  not  as  able  to  demonstrate  their  knowledge  and 
skills  when  they  are  tested  in  French  as  when  they  are  tested  in  English. 

The  language  of  testing  results  are  also  of  interest  when  considering  appropriate 
standards  and  norms  against  which  to  compare  student  achievement  in  French 
Immersion  programs.  What  can  be  concluded  from  this  study  is  that  where  there  is  a 
depression  of  scores  as  a result  of  the  language  of  testing,  it  is  inappropriate  to  compare 
the  results  of  the  group  writing  in  that  language  against  a test  score  that  has  been  chosen 
to  represent  the  standard  on  the  test  of  the  other  language.  It  appears  to  be  equally 
inappropriate  to  compare  the  scores  of  groups  writing  in  French  with  those  achieved  by 
students  writing  in  English  where  there  is  a language  of  testing  effect,  unless  that  effect 
has  been  accounted  for  in  some  way. 

While  this  study  indicates  that  a more  accurate  measure  of  what  French  Immersion 
students  know  and  can  do  is  obtained  when  they  are  tested  in  English,  it  leaves 
unanswered  the  issue  of  whether  testing  them  in  English  or  in  French  is  better 
pedagogically  or  politically.  It  may  well  be  that  what  is  most  valid  and  reliable 
statistically  might  not  lend  the  best  support  to  what  happens  in  the  classroom. 
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FUTURE  DIRECTIONS 


Student  Evaluation  staff  are  addressing  the  issues  raised  in  the  Language  of  Testing 
Study  in  1992.  First,  discussions  of  these  issues  have  been  started.  A cross-section  of 
educators  are  involved  in  these  discussions.  Second,  a final  report  addressing  these 
issues  and  summarizing  the  recommendations  arising  from  the  discussions  will  be 
prepared. 
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